Skip to content

fix(inference): show account status and fail over across hosts - #146

Merged
ScriptedAlchemy merged 5 commits into
mainfrom
fix/inference-proxy-account-status
Nov 17, 2026
Merged

ScriptedAlchemy merged 5 commits into
mainfrom
fix/inference-proxy-account-status

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

CLIProxyAPI can retain models in its catalog while hosted accounts are rejected or cooling down. T3 also kept one inference host per provider, so a 403, 429, or offline host stopped a remote thread even when another authorized environment could serve the same model.

Show the proxy's per-account availability in Inference accounts and filter unavailable route choices. Connected web and mobile clients discover other shared inference hosts, register them on the executing environment, and refresh discovery as host status changes. The local proxy keeps the requested model and execution environment, probes healthy hosts, retries a bounded request after auth, quota, or transport failures, combines healthy model catalogs, and keeps each thread on its successful host. Existing saved routes migrate to the driver-scoped local proxy path; reconnecting the same primary retains fallbacks. LAN addresses advertised by a host are verified before use.

Local and remote providers now get a stable forwarding address from their first launch. Background discovery respects explicit revocation until the user restores the grant. T3 identifies its own grant rejection in the response. OpenCode and Pi preserve native models, including project models, when a proxy account goes offline.

OpenCode owns retry timing while a turn runs. An explicit service-overloaded response, including HTTP 429, remains a provider error after OpenCode exhausts its retries. On genuine terminal rate-limit failures, the v2 adapter preserves the provider's retry duration and derives a reset time from the event timestamp. The v1 adapter reads the standard retry header when available. This lets the existing per-thread and environment resume controls schedule a continuation. Proxy forwarding handles both seconds and millisecond retry hints.

With the installed OpenCode 2.3.0 CLI and isolated loopback fixtures, both a token-limit 429 and a service-overload 429 produce terminal provider.rate-limit events. Their retained response bodies distinguish rate_limit_exceeded from service_overloaded, and the event includes retryAfterMs. The adapter regression tests cover that distinction.

The changed-file gate passed (568 tests, 42 affected package typechecks, format, lint, Knip, GraphQL, and React compilation). The mounted web coordinator test covers primary disconnect, alternative recovery, duplicate status updates and a primary change. Server tests cover scoped grants, revocation, self and LAN routes, proxy model catalogs and native OpenCode/Pi catalog recovery. The historical Claude failures reached the SDK as bodyless 403s across two Studio threads and two models. The retained events do not identify the responding endpoint or account. A real LAN request previously succeeded. In a separate controlled LAN run, T3 relayed the same Claude request after a bodyless 403 to the second host and used that host again for the next request. Deployment and an installed-client fleet cutover remain unverified.

Account affinity is per CLIProxyAPI host. T3 forwards session identity and holds a successful host for a thread, but switching between hosts with multiple eligible accounts cannot guarantee the same account without a shared account-selection contract. Requests above the 8 MiB replay bound stream once through the selected healthy host and cannot retry after a downstream auth or quota response.

Hosted Claude refresh also checks the subscription usage endpoint through CLIProxyAPI for each eligible account. A confirmed exhausted account-wide window supplies a reset time and suppresses the account while the limit is in effect. Slow or denied usage reads leave the account eligible for an inference request. Mixed pools consult account model registrations and cooldowns so another active account can keep its own models available. A live subscription usage read succeeded while two inference models returned 429 with a matching retry delay.

Models: GPT-6.2-Sol and GPT-6.2-Astra. Harness: Codex.

Read CLIProxyAPI account health when refreshing provider routes and show safe per-account status in settings. Reject new routes when no hosted account is eligible.
@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bd4c7b5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review November 17, 2026 19:17
@ScriptedAlchemy ScriptedAlchemy changed the title fix(inference): show hosted account availability fix(inference): show account status and fail over across hosts Nov 17, 2026
@ScriptedAlchemy
ScriptedAlchemy merged commit 9d28a97 into main Nov 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant